Add Top-N pushdown support for the MongoDB connector #26504

codluca · 2025-08-28T07:52:16Z

Description

Add Top-N (ORDER BY + LIMIT) pushdown support for the MongoDB connector.

Additional context and related issues

MongoDB considers null values to be less than any other value. We can handle sort items with SortOrder.ASC_NULLS_LAST or SortOrder.DESC_NULLS_FIRST with a MongoDB aggregation pipeline, where we add computed fields to sort correctly.

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
(X ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

## Section
* Implement feature. ({issue}`26366`)

codluca · 2025-08-29T15:58:18Z

The pull request is ready for review. Thanks!

chenjian2664 · 2025-08-31T08:24:16Z

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoMetadata.java

    }

+    @Override
+    public Optional<TopNApplicationResult<ConnectorTableHandle>> applyTopN(ConnectorSession session, ConnectorTableHandle table, long topNCount, List<SortItem> sortItems, Map<String, ColumnHandle> assignments)


Please put each variable on a new line

Thanks for the review!

Fixed.

chenjian2664 · 2025-08-31T10:23:17Z

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoMetadata.java

+            sortDocument.append(columnName, direction);
+        }
+        List<MongoTableSort> tableSortList = handle.sort().orElse(new ArrayList<>());
+        MongoTableSort tableSort = new MongoTableSort(sortDocument, sortNullFieldsDocument == null ? Optional.empty() : Optional.of(sortNullFieldsDocument), toIntExact(topNCount));


We could use Optional.ofNullable

chenjian2664 · 2025-08-31T10:26:24Z

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoMetadata.java

+            if (sortItem.getSortOrder() == SortOrder.ASC_NULLS_LAST || sortItem.getSortOrder() == SortOrder.DESC_NULLS_FIRST) {
+                String sortColumnName = "_sortNulls_" + columnName;
+                Document condition = new Document();
+                condition.append("$cond", List.of(new Document("$eq", new ArrayList<>(Arrays.asList("$" + columnName, null))), 1, 0));


Suggested change

condition.append("$cond", List.of(new Document("$eq", new ArrayList<>(Arrays.asList("$" + columnName, null))), 1, 0));

condition.append("$cond", ImmutableList.of(new Document("$eq", Arrays.asList("$" + columnName, null)), 1, 0));

chenjian2664 · 2025-08-31T10:27:06Z

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoMetadata.java

+        Document sortNullFieldsDocument = null;
+        for (SortItem sortItem : sortItems) {
+            String columnName = sortItem.getName();
+            int direction = (sortItem.getSortOrder() == SortOrder.ASC_NULLS_FIRST || sortItem.getSortOrder() == SortOrder.ASC_NULLS_LAST) ? 1 : -1;


can we give 1 and -1 a meaningful name?

chenjian2664 · 2025-08-31T10:28:24Z

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoTableSort.java

+                .toList();
+        builder.append(", orderBy = ").append(sortEntries);
+        if (sortNullFields.isPresent()) {
+            List<String> nullFields = sortNullFields.get().keySet().stream()


Use ImmutableList.copyOf

chenjian2664 · 2025-08-31T10:29:11Z

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoSession.java

+                for (String sortField : tableSort.sort().keySet()) {
+                    // Sorting on the field does not work unless we add it to the projection
+                    if (!projection.containsKey(sortField)) {
+                        projection.append(sortField, 1);


Same here, it's not obvious for the meaning of 1

Praveen2112 · 2025-09-06T16:19:06Z

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoMetadata.java

+    public Optional<TopNApplicationResult<ConnectorTableHandle>> applyTopN(
+            ConnectorSession session,
+            ConnectorTableHandle table,
+            long topNCount,


What if the topNCount is more than range of integer ?

Added check similar to applyLimit method

// MongoDB doesn't support topN number greater than integer max if (topNCount > Integer.MAX_VALUE) { return Optional.empty(); }```

Praveen2112 · 2025-09-06T16:19:52Z

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoMetadata.java

+            MongoColumnHandle columnHandle = (MongoColumnHandle) columnHandleEntry.getValue();
+            if (!columnHandleEntry.getKey().equals(columnHandle.baseName())) {
+                // We don't support complex nested queries
+                return Optional.empty();


How about for dbRef column type ?

As I understand it, applyTopN is called only for one table, no JOIN. The dbRef from MongoDB would be a reference to another collection, what we achieve with a JOIN. If it's something else, please clarify.
I wrote a simple test, and the execution doesn't enter the applyTopN method.

* Test topN dbRef. */ @Test public void testJoinTopNDbRef() { Session session = Session.builder(getSession()) .setSystemProperty(IGNORE_STATS_CALCULATOR_FAILURES, "false") .build(); assertQuery( session, "SELECT n.name, c.name " + "FROM nation n " + "JOIN customer c ON c.nationkey = n.nationkey " + "ORDER BY n.nationkey, c.name LIMIT 2"); }```

If it is the case, we could simply add a defense check to Make sure the columnHandle#dbRefField is false, otherwise return Optional.empty() to indicate we don't support it

AbstractTestQueries.testPredicate() fails.

@Test public void testPredicate() { assertQuery("" + "SELECT *\n" + "FROM (\n" + " SELECT orderkey+1 AS a FROM orders WHERE orderstatus = 'F' UNION ALL \n" + " SELECT orderkey FROM orders WHERE orderkey % 2 = 0 UNION ALL \n" + " (SELECT orderkey+custkey FROM orders ORDER BY orderkey LIMIT 10)\n" + ") \n" + "WHERE a < 20 OR a > 100 \n" + "ORDER BY a"); }

columnHandle.dbRefField is false.
However,
columnHandle.baseName() is "orderkey"
columnHandleEntry.getKey() is "orderkey_12"
So I used this condition to avoid the pushdown for this test case also.

chenjian2664 · 2025-10-12T01:48:23Z

involve #26521, we should wait that to cover the basic test

github-actions · 2025-11-03T17:03:46Z

This pull request has gone a while without any activity. Ask for help on #core-dev on Trino slack.

chenjian2664 · 2025-11-06T01:55:45Z

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoMetadata.java

+            MongoColumnHandle columnHandle = (MongoColumnHandle) columnHandleEntry.getValue();
+            if (!columnHandleEntry.getKey().equals(columnHandle.baseName())) {
+                // We don't support complex nested queries
+                return Optional.empty();


If it is the case, we could simply add a defense check to Make sure the columnHandle#dbRefField is false, otherwise return Optional.empty() to indicate we don't support it

chenjian2664 · 2025-11-06T02:06:09Z

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoTableHandle.java

        Optional<String> filter,
        TupleDomain<ColumnHandle> constraint,
        Set<MongoColumnHandle> projectedColumns,
+        Optional<List<MongoTableSort>> sort,


Why it requires List instead of Optional

If a single MongoTableSort is used,
BaseConnectorTest.testTopNPushdown() test fails.

// TopN over TopN assertThat(query("SELECT orderkey, totalprice FROM (SELECT orderkey, totalprice FROM orders ORDER BY 1, 2 LIMIT 10) ORDER BY 2, 1 LIMIT 5")) .ordered() .isFullyPushedDown();

Expected :[[2, 38426.09], [6, 45523.1], [4, 56000.91], [34, 73315.48], [5, 105367.67]] Actual :[[35271, 874.89], [28647, 924.33], [58145, 929.03], [8354, 974.04], [37415, 986.63]]

Also, AbstractTestQueries.testTopN() fails for similar issue.

// TopN over TopN assertQueryOrdered("SELECT orderkey, totalprice FROM (SELECT orderkey, totalprice FROM orders ORDER BY 1, 2 LIMIT 10) ORDER BY 2, 1 LIMIT 5");

The nested SELECT will produce a MongoTableSort.
The root SELECT will need to aggregate the MongoTableSort entities.

~~oh, got it, so it is not "entire table sort", it is actual one sort item, we could rename it similar to "MongoSortItem"~~
It seems not, I needs to take a look at this

So this is for the case handle push topN into a table already contains topN info

Yes.
Should I do the rename to MongoSortItem?

@chenjian2664
Yes.
Should I do the rename to MongoSortItem?

Add Top-N (ORDER BY + LIMIT) pushdown support for the MongoDB connector. MongoDB considers null values to be less than any other value. We can handle sort items with SortOrder.ASC_NULLS_LAST or SortOrder.DESC_NULLS_FIRST with a MongoDB aggregation pipeline, where we add computed fields to sort correctly. We add sort fields to the projection, so that they are considered by MongoDB. We add Top-N pushdown support only for simple nested queries (that work on the same collection).

chenjian2664 · 2025-11-07T09:53:39Z

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoMetadata.java

+
+            sortDocument.append(columnName, direction);
+        }
+        List<MongoTableSort> tableSortList = handle.sort().orElse(new ArrayList<>());


Please consider using ImmutableList.Builder

chenjian2664 · 2025-11-07T09:54:39Z

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoMetadata.java

+                        handle.constraint(),
+                        handle.projectedColumns(),
+                        Optional.of(tableSortList),
+                        OptionalInt.empty()),


It's safer to put handle.limit()

chenjian2664 · 2025-11-07T09:55:04Z

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoSession.java

+            List<Document> aggregateList = new ArrayList<>();
+            aggregateList.add(new Document("$match", filter));
+            for (MongoTableSort tableSort : tableSortList) {
+                tableSort.sortNullFields().ifPresent(sortNullFields -> new Document("$addFields", sortNullFields));


This line seems not doing anything?

chenjian2664 · 2025-11-07T10:05:09Z

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoSession.java

+                }
+            }
+
+            List<Document> aggregateList = new ArrayList<>();


use ImmuableList and rename the aggregateList to aggregatePipeline

cla-bot bot added the cla-signed label Aug 28, 2025

github-actions bot added the mongodb MongoDB connector label Aug 28, 2025

codluca requested a review from ebyhr August 28, 2025 07:53

ebyhr requested a review from chenjian2664 August 28, 2025 07:57

codluca force-pushed the 26366 branch from 60a2d2f to 6d49803 Compare August 29, 2025 09:27

codluca marked this pull request as draft August 29, 2025 10:08

codluca force-pushed the 26366 branch 2 times, most recently from 8973aee to 0331d72 Compare August 29, 2025 13:27

codluca marked this pull request as ready for review August 29, 2025 15:57

codluca mentioned this pull request Aug 29, 2025

Top-N pushdown support for the MongoDB connector #26366

Open

codluca force-pushed the 26366 branch from 0331d72 to f7d5300 Compare August 29, 2025 19:00

chenjian2664 reviewed Aug 31, 2025

View reviewed changes

codluca force-pushed the 26366 branch 2 times, most recently from ea62cea to f1754ea Compare August 31, 2025 20:42

Praveen2112 reviewed Sep 6, 2025

View reviewed changes

codluca force-pushed the 26366 branch from 945d692 to c69402e Compare September 18, 2025 09:10

codluca force-pushed the 26366 branch from c69402e to 88984d0 Compare October 7, 2025 12:42

github-actions bot added the stale label Nov 3, 2025

chenjian2664 reviewed Nov 6, 2025

View reviewed changes

codluca force-pushed the 26366 branch from 88984d0 to 8b299fa Compare November 6, 2025 09:26

codluca force-pushed the 26366 branch from 9cecd49 to cc11185 Compare November 6, 2025 11:13

github-actions bot removed the stale label Nov 6, 2025

chenjian2664 reviewed Nov 7, 2025

View reviewed changes

	condition.append("$cond", List.of(new Document("$eq", new ArrayList<>(Arrays.asList("$" + columnName, null))), 1, 0));
	condition.append("$cond", ImmutableList.of(new Document("$eq", Arrays.asList("$" + columnName, null)), 1, 0));

Add Top-N pushdown support for the MongoDB connector #26504

Are you sure you want to change the base?

Add Top-N pushdown support for the MongoDB connector #26504

Conversation

codluca commented Aug 28, 2025

Description

Additional context and related issues

Release notes

Uh oh!

codluca commented Aug 29, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codluca Sep 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chenjian2664 commented Oct 12, 2025

Uh oh!

github-actions bot commented Nov 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codluca Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chenjian2664 Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

codluca Sep 6, 2025 •

edited

Loading

codluca Nov 6, 2025 •

edited

Loading

chenjian2664 Nov 7, 2025 •

edited

Loading